1. Field of the Invention
This invention pertains in general to computer security and in particular to detecting attempted exploits of vulnerabilities of applications and other programs executing on a computer.
2. Description of the Related Art
Applications executed on modern computers are often susceptible to a wide variety of network-based attacks. Web browsers, for example, are particularly susceptible to attacks because browsers receive large amounts of content from the Internet. The content can include hypertext markup language (HTML) web pages, scripts in languages such as VBScript and JavaScript, and executable content. Other types of applications are also vulnerable. For example, email programs and even word processors provide interfaces for executing network-based content.
Malicious attackers can compromise such applications by crafting specially-formulated input that exploits vulnerabilities in the programs. This input contains code that, when executed, gives the attackers control over the applications and allows them to perform malicious acts such as capturing keystrokes, sending messages on the network, deleting files, installing malicious software (malware) such as spyware and adware, etc.
One technique for detecting and preventing these malicious exploits is scanning network traffic entering the computer in order to detect malicious code. For example, an intrusion detection system (IDS) can scan network traffic entering an enterprise network for characteristics of malicious code, and then prevent the code from entering the enterprise. A problem with this approach is that attackers can use classical obfuscation and encryption techniques to evade detection. For example, an attacker can create a seemingly-innocent script that passes through the IDS but produces a malicious script when executed by an application.
The classical response to detecting obfuscated malicious code is to emulate the seemingly-innocent code to determine whether it produces malicious code. However, emulation is difficult to perform in this context. Oftentimes, there are many scripts and other types of code simultaneously passing through the IDS, or even entering a single computer, and the resources required to emulate all of the code are not available. Moreover, the code can incorporate techniques to defeat emulation, such as using lengthy delays before producing the malicious code or detecting the emulator and altering the malicious behavior.